Preserve existing metadata labels (and other fields) in SparkApplication manifest #56063

mohamed-mis · 2025-09-24T16:00:23Z

This PR addresses an issue in the CustomObjectLauncher operator within the Kubernetes provider, where submitting a SparkApplication manifest -using the SparkKubernetesOperator- results in the loss of existing metadata labels and annotations.

The current implementation in get_body() overwrites the entire metadata object:
self.body.metadata = {"name": self.name, "namespace": self.namespace}
This discards any pre-existing metadata such as labels, annotations, or custom fields.
This PR replaces the overwrite with an in-place update using dict.update().

Testing:
This fix has been validated with the following setup:

Airflow Version: 3.0.6
Deployment Method: Official Helm chart 1.18.0
Kubernetes Provider Version: apache-airflow-providers-cncf-kubernetes==10.7.0
Python Version: 3.12.11

SparkApplication resources submitted via the SparkKubernetesOperator now retain their original metadata labels as expected.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Ensure that CustomObjectLauncher updates only the 'name' and 'namespace' fields in the metadata without overwriting existing labels or annotations. This change uses dict.update() to safely modify the metadata in place, preventing loss of important SparkApplication metadata during submission.

boring-cyborg · 2025-09-24T16:00:27Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide (https://github.com/apache/airflow/blob/main/contributing-docs/README.rst)
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack

mohamed-mis · 2025-09-26T10:17:14Z

All GHA checks were successful, except for [CI image checks / Static checks], which failed due missing licence text in other files not related to the PR.

nevcohen

Looks good! Just add test

mohamed-mis · 2025-09-29T09:50:53Z

tests for the custom_object_launcher already exist and covers the updated part.
test_custom_object_launcher.py

eladkal · 2025-09-29T10:51:57Z

tests for the custom_object_launcher already exist and covers the updated part.

Your PR changes/fix behavior thus you can't claim tests already cover the fix.
You don't have to write a new test, it could be that extend or fix to a current test is enough. The idea is that the change you are implemented here won't be regressed in future versions so the way to make sure it doesn't happen is with test coverage.

test suite for get_body logic in CustomObjectLauncher, including metadata handling

mohamed-mis · 2025-09-29T21:43:52Z

Tests added:

test_get_body_initializes_metadata_when_missing
test_get_body_replaces_non_dict_metadata
test_get_body_preserves_existing_metadata_labels

Waiting for GHA to finish all workflows, otherwise it should be good (fingers Crossed).

nevcohen · 2025-09-30T15:18:36Z

Now it's ready. It looks great!

boring-cyborg · 2025-09-30T15:50:29Z

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

…plication manifest (apache#56063) * Preserve existing metadata labels in SparkApplication manifest Ensure that CustomObjectLauncher updates only the 'name' and 'namespace' fields in the metadata without overwriting existing labels or annotations. This change uses dict.update() to safely modify the metadata in place, preventing loss of important SparkApplication metadata during submission. * misc: format committed lines for consistency * Fix: Safely handle missing `metadata` attribute in SparkJobSpec * Add unit tests for get_body method in CustomObjectLauncher test suite for get_body logic in CustomObjectLauncher, including metadata handling

mohamed-mis requested review from hussein-awala and jedcunningham as code owners September 24, 2025 16:00

boring-cyborg bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Sep 24, 2025

mohamed-mis added 3 commits September 25, 2025 10:33

misc: format committed lines for consistency

1ef82d7

Merge branch 'main' into patch-2

640fd3e

Fix: Safely handle missing metadata attribute in SparkJobSpec

bcb2ed4

nevcohen approved these changes Sep 28, 2025

View reviewed changes

Add unit tests for get_body method in CustomObjectLauncher

8af38fe

test suite for get_body logic in CustomObjectLauncher, including metadata handling

Merge branch 'main' into patch-2

58d32d0

eladkal approved these changes Sep 30, 2025

View reviewed changes

eladkal merged commit 422c6eb into apache:main Sep 30, 2025
90 checks passed

eladkal mentioned this pull request Oct 22, 2025

Status of testing Providers that were prepared on October 22, 2025 #57080

Closed

69 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Preserve existing metadata labels (and other fields) in SparkApplication manifest #56063

Preserve existing metadata labels (and other fields) in SparkApplication manifest #56063

Uh oh!

mohamed-mis commented Sep 24, 2025

Uh oh!

boring-cyborg bot commented Sep 24, 2025

Uh oh!

mohamed-mis commented Sep 26, 2025 •

edited

Loading

Uh oh!

nevcohen left a comment •

edited

Loading

Uh oh!

mohamed-mis commented Sep 29, 2025

Uh oh!

eladkal commented Sep 29, 2025

Uh oh!

mohamed-mis commented Sep 29, 2025

Uh oh!

nevcohen commented Sep 30, 2025

Uh oh!

Uh oh!

boring-cyborg bot commented Sep 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Preserve existing metadata labels (and other fields) in SparkApplication manifest #56063

Preserve existing metadata labels (and other fields) in SparkApplication manifest #56063

Uh oh!

Conversation

mohamed-mis commented Sep 24, 2025

Uh oh!

boring-cyborg bot commented Sep 24, 2025

Uh oh!

mohamed-mis commented Sep 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nevcohen left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mohamed-mis commented Sep 29, 2025

Uh oh!

eladkal commented Sep 29, 2025

Uh oh!

mohamed-mis commented Sep 29, 2025

Uh oh!

nevcohen commented Sep 30, 2025

Uh oh!

Uh oh!

boring-cyborg bot commented Sep 30, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mohamed-mis commented Sep 26, 2025 •

edited

Loading

nevcohen left a comment •

edited

Loading